Characterizing Movement Fluency in Musical Performance: Toward a Generic Measure for Technology Enhanced Learning

Virtuosity in music performance is often associated with fast, precise, and efficient sound-producing movements. The generation of such highly skilled movements involves complex joint and muscle control by the central nervous system, and depends on the ability to anticipate, segment, and coarticulate motor elements, all within the biomechanical constraints of the human body. When successful, such motor skill should lead to what we characterize as fluency in musical performance. Detecting typical features of fluency could be very useful for technology-enhanced learning systems, assisting and supporting students during their individual practice sessions by giving feedback and helping them to adopt sustainable movement patterns. In this study, we propose to assess fluency in musical performance as the ability to smoothly and efficiently coordinate while accurately performing slow, transitionary, and rapid movements. To this end, the movements of three cello players and three drummers at different levels of skill were recorded with an optical motion capture system, while a wireless electromyography (EMG) system recorded the corresponding muscle activity from relevant landmarks. We analyzed the kinematic and coarticulation characteristics of these recordings separately and then propose a combined model of fluency in musical performance predicting music sophistication. Results suggest that expert performers' movements are characterized by consistently smooth strokes and scaling of muscle phasic coactivation. The explored model of fluency as a function of movement smoothness and coarticulation patterns was shown to be limited by the sample size, but it serves as a proof of concept. Results from this study show the potential of a technology-enhanced objective measure of fluency in musical performance, which could lead to improved practices for aspiring musicians, instructors, and researchers

Gestural-Sonorous Objects: embodied extensions of Schaeffer’s conceptual apparatus

One of the most remarkable achievements of Pierre Schaeffer's musical thought is his proposal of the sonorous object as the focus of research. The sonorous object is a fragment of sound, typically in the range of a few seconds (often even less), perceived as a unit. Sonorous objects are constituted, studied, and evaluated according to various criteria, and sonorous objects that are found suitable are regarded as musical objects that may be used in musical composition. In the selection and qualification of these sonorous objects, we are encouraged to practise what Schaeffer called ‘reduced listening’, meaning disregarding the original context of the sound, including its source and signification, and instead focus our listening on the sonorous features. However, it can be argued that this principle of ‘reduced listening’ is not in conflict with more fundamental principles of embodied cognition, and that the criteria for the constitution, and the various feature qualifications, of sonorous objects can be linked to gestural images. Also, there are several similarities between studying sound and gestures from a phenomenological perspective, and it is suggested that Schaeffer's theoretical concepts may be extended to what is called gestural-sonorous objects

Geometry and Effort in Gestural Renderings of Musical Sound

As may be seen at concerts and in various everyday listening situa- tions, people often make spontaneous gestures when listening to music. We believe these gestures are interesting to study because they may reveal important features of musical experience. In particular, hand movements may give us information on what features are perceived as salient by listeners. Based on various current ideas on embodied cognition, the aim of this paper is to argue that gestures are integral to music perception, and to present research in support of this. A conceptual model of separating geometry and effort is presented in order to better understand the variety of music-related gestures we may observe, lead- ing up to some ideas on how to apply this conceptual model in present and future research. The final publication is available at link.springer.com

Understanding Coarticulation in Music

The term coarticulation designates the fusion of small-scale events, such as single sounds and single sound-producing actions, into larger units of combined sound and body motion, resulting in qualitative new features at what we call the chunk timescale in music, typically in the 0.5.–5 s duration range. Coarticulation has been extensively studied in linguistics and to a certain extent in other domains of human body motion as well as in robotics, but so far not so much in music, so the main aim of this paper is to provide a background for how we can explore coarticulation in both the production and perception of music. The contention is that coarticulation in music should be understood as based on a number of physical, biomechanical and cognitive constraints, and that coarticulation is an essential factor in the shaping of several perceptually salient features of music

Images of Sonic Objects

Based on innumerable informal accounts and a number of scientific studies, there can be no doubt that people often have quite vivid images of musical sound in their minds, and that this is the case regardless of levels of musical training. Various introspective accounts and more recent neurocognitive research seem to converge in suggesting that imagery for music is closely linked with imagery for music-related actions. In this paper, the consequences of sound–action links for our notions of the sonic image are discussed, with a particular focus on the relationship between sonic objects and action chunks. In conclusion, the exploitation of action imagery is seen as holding great promise in enhancing our means for musical imagery in various creative, research and educational contexts

Understanding Musical Instants

We may typically experience music as continuous streams of sound and associated body motion, yet we may also perceive music as sequences of more discontinuous events, or as strings of chunks with multimodal sensations of sound and body motion, chunks that can be called ‘sound-motion objects’. The focus in this chapter is on how such sound-motion objects emerge at intermittent points in time called ‘musical instants’, and how musical instants are necessary in order to perceive salient features in music such as of timbre, pitch, texture, contour, and overall stylistic and affective features. The emergence of musical instants is also understood as based on the combined constraints of musical instruments, sound-producing body motion, and music perception, also suggesting that understanding musical instants may have practical applications in making music

Perceiving Sound Objects in the Musique Concrète

In the late 1940s and early 1950s, there emerged a radically new kind of music based on recorded environmental sounds instead of sounds of traditional Western musical instruments. Centered in Paris around the composer, music theorist, engineer, and writer Pierre Schaeffer, this became known as musique concrète because of its use of concrete recorded sound fragments, manifesting a departure from the abstract concepts and representations of Western music notation. Furthermore, the term sound object was used to denote our perceptual images of such fragments. Sound objects and their features became the focus of an extensive research effort on the perception and cognition of music in general, remarkably anticipating topics of more recent music psychology research. This sound object theory makes extensive use of metaphors, often related to motion shapes, something that can provide holistic representations of perceptually salient, but temporally distributed, features in different kinds of music

Motormimetic features in musical experience

There can be no doubt that we often experience correspondences between different sense modalities in music, such as between sound, vision, motion, and touch (just to mention the most prominent ones). This is evident in dance and other kinds of music-related body motion, and also reflected in listeners' innumerable accounts of visual associations with music, and in the ubiquitous use of visual metaphors for musical sound such as "rough", "smooth", "narrow", "broad", etc. In short, it should not be controversial to say that music is a multimodal form of art, that music involves a number of sensations in addition to pure sound. But more precisely how different sense modalities are activated, and how they interact in musical experience, still presents us with a number of unanswered questions. In our research, we have been pursuing the idea of what we call "motormimetic cognition", meaning an incessant mental simulation of sound-related body motion in music perception, primarily of assumed sound-producing body motion (e.g. hitting, stroking, bowing, blowing), but also of various kinds of sound-accompanying body motion (e.g. dancing, walking, gesticulating). We regard such mental simulation of body motion as applicable to most features of music, and we believe motormimetic cognition also can translate between different modalities, for instance that mental simulation of motion can translate between sound and visual images in musical experience. We thus believe motormimetic cognition can be the basis for a systematic research effort on feature mapping between different modalities in music, by way of studying both musicians' and listeners' body motion trajectory shapes and/or posture shapes in music-related contexts. In this chapter, we shall present basic principles of motormimetic cognition and demonstrate how it is relevant for work with new technologies and in multimedia art. Starting out with a review of timescales in our experiences of sound and motion, we shall present a rudimentary taxonomy of sound-motion categories, proceeding in a top-down manner from some global, salient features down to a fairly large number of detail features. This should enable us to have a versatile and flexible conceptual apparatus, as well as a collection of practical tools (using available technologies for sound and motion research), that will allow us to explore our aesthetic and affective images of sound and motion in music, hopefully also contributing to bridging the gaps between quantitative and qualitative approaches in research. © 2017 Routledg

Timescales for sound-motion objects

39481